A Dynamic Programming Approach To Document Clustering Based On Term Sequence Alignment

Document clustering is unsupervised machine learning technique that, when provided with a large document corpus, automatically sub-divides it into meaningful smaller sub-collections called clusters. Currently, document clustering algorithms use sequence of words (terms) to compactly represent documents and define a similarity function based on the sequences. We believe that the word sequence is vital in determining the contextual information of a document. A frequent sequence, maximal sequence or multi-word sequence cannot alone give good contextual information. These sequences can affect the local similarity computation and can compromise the accuracy of the final clusters. Motivated by this, we propose in this paper a dynamic programming approach to document clustering based on term sequence alignment. There are three main contribution through this research: (i) a document representation model is proposed based on sequence of term used, (ii) a similarity measure is defined that uses term sequence alignment score to assign relatedness for a pair of documents, and (iii) a dynamic programming based hierarchical agglomerative clustering (HAC) algorithm is proposed to cluster the documents. Moreover, the closely related works, (a) Frequent Itemset-based Hierarchical Clustering (FIHC) and (b) Text document clustering based on frequent word meaning sequences (CFWS) are extensively evaluated in comparison of proposed algorithm, on classical text mining datasets. The proposed algorithm significantly improves the quality of the clusters produced and is comparable to state-of-the-art text/document clustering algorithms..